Importance:

• Reproduction and critical assessment
• Erroneous results and misinformation
• Standardization

Many aspects already studied in (among others) psychology, but not in sociology.

1. Requested adherence to statistical reporting guidelines (i.e., APA manual) by journals
2. Statistical reporting errors
   • inconsistencies
   • gross inconsistencies
3. Publication bias/p-hacking
   • replication Gerber and Malhotra (2008)
4. ‘Bump’ in p-values
5. Marginal significance

Note that not all topics are (fully) addressed in this presentation.

• Data set with information on which journals require adherence to which guidelines
  • all 143 sociology journals from Clarivate Analytics’ Web of Science (2016) were included
• Data set AllP
  • automatic retrieval
  • all p-values retrieved using Epskamp and Nuijten (2016)’s statcheck
  • information on marginal significance

• Data set APA
  • automatic retrieval
    
  • all APA-reported results as retrieved using statcheck
• Data set Hyp
  • Created to replicate Gerber and Malhotra (2008)
  • manual retrieval
  • all p-values of hypotheses
  • all reproducible results of hypotheses
  • information on marginal significance for results of hypotheses

For JMF, statcheck could initially not extract results with a minus sign. One of the package’s authors solved this problem for us by adapting the GitHub version of the package, which we installed using the R code below.

install.packages("devtools")
devtools::install_github("MicheleNuijten/statcheck")
install.packages("statcheck")

• For all topics: frequencies & percentages
• Logistic regressions for testing hypotheses
  
  

\[\scriptsize \begin{align} H1: log(Inconsistency_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \\ H2: log(Gross\,inconsistency_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \\ H3: log(Publication\,bias_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \\ H4: log(Marginal\,significance_{i}) &= \beta_0 + \beta_1\times hypothesis_i + \epsilon_i \end{align} \]

13 of 143 journals (9.1%) requested authors to adhere to APA statistical reporting guidelines